Philip Tetlock

Philip Tetlock is presented as the psychologist who gave the modern discussion of expert prediction its decisive empirical foundation. His work challenged the comforting image of the expert as someone whose specialized knowledge reliably reveals the future: across thousands of forecasts, professional commentators performed worse than a simple equal-probability baseline, while even the better-performing style of forecaster remained far from dependable.

Identity and Intellectual Position

Philip Tetlock was a psychologist at the University of Pennsylvania whose research focused on people who made their living commenting on political and economic trends. His importance in the narrative lies less in a personal biography than in the standards he imposed on public expertise. Instead of treating eloquence, reputation, or specialized knowledge as evidence of predictive power, he recorded forecasts, assigned them explicit probabilities, and compared their results with simple alternatives.

Tetlock’s landmark twenty-year study was published in his 2005 book Expert Political Judgment: How Good Is It? How Can We Know? The study became a reference point for later discussions of forecasting because it examined not merely what experts predicted, but how they reasoned, how they handled contrary evidence, and how they responded after being shown to be wrong.

The Forecasting Study

Tetlock interviewed 284 people who earned their living by commenting on or advising others about political and economic trends. He asked them to estimate the probabilities of events in the relatively near future, both in regions they knew well and in areas where they had less knowledge. The questions included whether Mikhail Gorbachev would be removed in a coup, whether the United States would go to war in the Persian Gulf, and which country would become the next major emerging market.

The project accumulated more than 80,000 predictions. For each question, experts assessed three possible directions: continuation of the status quo, an increase in a condition such as political freedom or economic growth, or a decrease in it. Tetlock also collected information about the experts’ methods, their reactions to disconfirming evidence, and the explanations they offered when their predictions failed.

The results were devastating for the ordinary prestige of expertise. The professionals did worse than they would have done by distributing equal probabilities across the three possible outcomes. Even in the regions they understood best, they were not significantly better than nonspecialists. Greater knowledge produced only a very slight improvement, and the most knowledgeable participants were often less reliable because knowledge strengthened their illusion of skill and encouraged unjustified confidence.

Tetlock found that fame made the problem worse. The most celebrated forecasters made more flamboyant predictions and were more overconfident than colleagues working outside the limelight. Their public value was therefore partly theatrical: television producers preferred clear, forceful opinions, especially when two confident experts could attack each other from opposing sides, even though that performance had little connection with forecasting accuracy.

Hedgehogs, Foxes, and Error

Tetlock adopted the distinction between hedgehogs and foxes from Isaiah Berlin’s discussion of Leo Tolstoy. Hedgehogs organize events around one large idea. They interpret particular developments through a coherent theory, speak with certainty, show impatience toward contrary views, and resist admitting error. When a prediction fails, a hedgehog commonly explains that it was wrong only in its timing, that an unforeseeable event intervened, or that the reasoning was correct even though the outcome was not.

Foxes think in a more plural and conditional way. They distrust explanations that reduce history to a single driving force, recognize the interaction of many agents and causes, and allow blind luck to produce large, unpredictable consequences. In Tetlock’s study, foxes performed better than hedgehogs. Their advantage was real but limited: even the more flexible style remained generally poor at forecasting, and foxes were less attractive to television because complexity and uncertainty make weaker entertainment than confident confrontation.

Tetlock’s findings therefore did not simply replace one class of experts with another. They showed that intellectual modesty can improve prediction without making the world predictable. The fox is preferable to the hedgehog, but neither style abolishes the uncertainty built into political and economic events.

Calibration and the Limits of Judgment

The broader forecasting problem concerns calibration: whether stated confidence corresponds to the frequency with which predictions come true. A properly calibrated judge who assigns a particular confidence level should find that the predicted proportion of outcomes falls within that range over a sufficiently large set of cases. Tetlock’s research made this standard central to the evaluation of public prediction, replacing retrospective admiration with records that can be checked against outcomes.

The same concern appears in research associated with Paul Meehl, whom the narrative identifies as one of Tetlock’s important intellectual predecessors in the comparison of expert judgment with formulas. Meehl reviewed twenty studies in which trained professionals were compared with statistical rules. In a typical example, counselors interviewed freshmen for forty-five minutes and consulted high-school grades, aptitude tests, and a four-page personal statement, while the formula used only high-school grades and one aptitude test. The formula was more accurate than eleven of fourteen counselors.

Later research expanded the comparison to roughly two hundred studies. About 60 percent showed significantly greater accuracy for algorithms; the remaining studies were ties, which still favored formulas because they were usually cheaper than expert judgment. The outcomes covered parole violations, pilot-training success, criminal recidivism, cancer-patient longevity, hospital stays, cardiac diagnoses, business success, credit risk, foster-parent suitability, violent behavior, football results, and Bordeaux wine prices. These were described as low-validity environments, in which the relationship between available information and future outcomes is weak or unstable.

Tetlock’s work is reinforced by the finding that human judgments are often inconsistent even when the evidence is unchanged. Experienced radiologists evaluating the same chest X-ray as normal or abnormal contradicted themselves 20 percent of the time on separate occasions, and a study of 101 auditors found a comparable problem. Statistical rules return the same result for the same input; human judgments are affected by context, priming, mood, and the unnoticed conditions under which the decision is made.

Legacy and Significance

Tetlock’s contribution is a disciplined skepticism toward the social performance of expertise. His study showed that confidence, fame, specialized knowledge, and vivid explanation are not reliable substitutes for a forecasting record. It also clarified why experts can remain persuasive after failure: they are often able to reinterpret errors as timing mistakes, interruptions by unforeseeable events, or near successes, preserving the story of competence even when the outcome contradicts it.

His work also connects forecasting with the planning fallacy and the outside view. Project teams commonly rely on the particular details and intentions visible from inside a case, while neglecting statistical information about comparable cases. In the curriculum project described in the narrative, even a group that had explicitly learned the outside view collectively ignored it when estimating how long its textbook would take. Tetlock’s research helps explain why such neglect persists: personal impressions create a compelling story, whereas base rates and reference classes appear pallid by comparison.

The lasting lesson is not that experts are worthless or that every formula is automatically wise. It is that prediction must be tested in the world it claims to understand. When outcomes are recorded, probabilities are explicit, and performance is compared with simple baselines, the authority of expertise becomes an empirical question. Tetlock’s foxes succeed because they acknowledge complexity and luck; they do not succeed because they have discovered a secret path through an unpredictable history.